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Introduction 



When it comes to knowing what is best for themselves and their students, there are 
no better experts than teachers. Their day-to-day classroom observations and activities 
constitute a wealth of valid and reliable research data, this extensive first-hand experience 
makes teachers the best craftspersons to design their own surveys., tests and other 
instrumentation which is ideally suited to their students, classrooms, and school situations. 

Despite such natural expertise, the fau is that teachers have tended to shy away 
from developing their own instrumentation. Part of the reason may be the illusory precision 
of nationally produced and marketed test packages. "Big-name recognition" and impressive 
packaging are admittedly as effective in educational marketing as in other selling endeavors. 

The irony, however, is that such slick outward packaging is no substitute for (and 
can actually impair) the usefulness of the product inside. All the impressive formatting in 
the world cannot compensate for the fact that a given test instrument may have been 
developed and normed on a group of students totally unlike one's own. Dereshiwsky 
(1992) has referred to this phenomenon as "the artificial child in Boston," one who may 
bear no resemblance to the bilingual, low socioeconomic rural school children with whom a 
teacher interacts every day in a local learning environment How useful is such a nationally 
marketed test in terms of assessing the needs and abilities of one's own children - here and 
now? The same point can be made about mass-marketed national surveys, and a school 
district's desire to study a given problem, situation or condition at the local level. 

The purpose of this paper is to present a procedure for developing and refining 
teacher-made survey instrumentation which is most valid and reliable for meeting local 
needs. (While a survey example will be used throughout this report, a similar argument 
applies to teacher-made tests.) First, a brief rationale will be given for teachers producing 
their own instrumentation, as opposed to automatically assuming that purchasing such 
materials from the outside is "better." Next, an easy-to-apply process for refining one's 
surveys and tests will be presented: one that requires no computers or statistics, but rather 
depends on little more than open sharing, discussion and communication which teachers do 
with their colleagues as a matter of course. To illustrate these procedures, the author will 
provide an actual example of a survey used to evaluate the 1992 Arizona Leadership 
Academy. 
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Tfr Thine Own Self Be True;" The Teacher as 'Ultimate 1 Survey Design Specialist 



As indicated in the preceding section, classroom teachers have the best possible 
vantage point for constructing locally appropriate surveys and tests. This is due to their 
extensive, day-to-day immersion in the situation and setting of the topic of the 
instrumentation. 

Unfortunately, there has been a tendency for such "grounded-theory experts" to 
distrust their own judgments when it comes to tests and surveys. Instead, there is an 
overreliance on mass-produced, slickly marketed national packages - which may be 
attractive and even easy to use, but which also may have little or nothing to do with the 
actual circumstances in which the teacher desires to apply them. 

Pan of thi s may have to do with the "mystique" of the research process, as well as a 
resulting fear and insecurity on the part of teachers. Dereshiwsky (1992) has identified two 
reasons for this avoidance. One has to do with " ... an unfortunate spillover effect from 
prior coursework experiences in statistics, computers and research design." The second 
stems from misconceptions such as, "It's too hard (corollary: 'You have to be a genius;' " 
"it takes too long (with images of Margaret Mead spending 20 years in a primitive 
location)"; and "it's only for college professors (or master's theses, or doctoral 
dissertations)." However, as later explained in the same paper: 

... the true, underlying purpose behind even the most sophisticated 
quantitative treatment is to answer an actual question ... and no more. To 
put it another way, without some 'real-life substance 1 behind it, number- 
crunching for its own sake is absolutely worthless ... Thus, in actuality, the 
whole process should begin with an idea, curiosity, or need to know - it's 
that simple ... the real star of the show is the research question. 
(Dereshiwskv, 1992) 

All measurements or observations, whatever the outward form or packaging of the 
research, must possess two qualities in order to be useful in answering the research 
question(s): they need to be valid and reliable. Validity has to do with credibility: am I 
measuring what I think I am? Reliability, on the other hand, deals with stability or 
consistency of the measurement. 

As mentioned earlier, the classroom teacher is in the ideal position to assess both of 
these properties in the measuring instrument which he/she needs to construct. This is 
because, due to his/her day-to-day involvement, the teacher has had the "longest look" at 
the individuals being measured or assessed - their special circumstances, needs, emotions, 
attitudes, and the like. Such qualitatively accepted "grounded-theory" immersion provides 
far more opportunity for valid and reliable observation of research needs than "single-shot" 
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experimental interventions, numeric number-crunched sole indicators of test quality, and so 
forth. 

This is not to imply that one needs to start from scratch and completely disregard 
other existing tests or surveys, literature reviews, and outside sources of information. Of 
course it is natuixl to rely upon existing work as a guide to one's own* The point is that 
there needs to be a balance and focus on the individual needs and circumstances for which 
the teacher is intending to use the test or survey. 

Fowler (1993) makes the following compelling argument for attaining such balance 
and trusting one's own judgment regarding locally appropriate instrumentation: 

Taking advantage of the work that others have done is very sensible. Of 
course, it is best to review questions asked by researchers who have done 
previous work on the study topic* In addition, if questions have been asked 
of other samples, collecting comparable data may add to the gcneralizability 
of the research* The mere feet that someone else has used a question before, 
however, is no guarantee that it is a very good question or, certainly, that it 
is an appropriate question for a given survey* Many bad questions are asked 
over and over again because researchers use them uncritically. All questions 
should be tested to mcke sure that they "work" for the populations, context 
and goals of a particular study, (pg* 97, emphasis mine) 

Figure 1 (below) graphically depicts the process and a few of the multiple sources 
of developing an initial draft of a test or survey for a locally based research question or 
need* 

.. > 

Figure 1. 

Factors to Consider in Designing Locally Appropriate Instrumentation 
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"Practice Makes Perfect:" T he Revision Process Made (Relative!^ Simple 



Having debunked at least some of the stereotypes of the research "mystique" in the 
previous section, it may be necessary to revisit at least one of them at this point. The fact is 
that no single piece of research, however painstakingly planned and refined, is ever 
"flawless" per se. This is not meant to imply that it is useless in answering the research 
question: far from it. Rather, some cautionary notes about its potential expansion, 
refinement and applicability to other settings and subjects are typically discussed in a 
section called "Limitations and Delimitations." 

This means that it would be rare for a first draft of a survey or test to be ideal - no 
matter how carefully the teacher may have prepared it, considering all of the sources of 
information in Figure 1. As with manuscripts and the like, it is wise to consider doing a 
"road test" of one's initial writing effort, in an attempt to find ways to improve it. In survey 
research, this process is known as pilot-testing. 

While sophisticated statistical procedures admittedly exist for pilot-test assessments 
of survey validity and reliability, it is important to note that they are not the only (or even 
the BEST way) for a teacher to choose to refine his/her locally developed instrument. One 
of the quickest, most time- and cost-efficient and naturally intuitive ways to do so is to 
consider what novice teachers writing a "first exam in ... " have done for years: run their 
draft by a handful of colleagues. The same procedure is used in survey research and is 
known as "qualitative expert-judge validation/ 1 It simply involves asking a small number of 
pilot subjects to review the instrumentation and to provide feedback as to its strengths, 
insufficiencies and recommendations for improvement. (Packard and Dereshiwsky have 
utilized these three qualities or dimensions in a variety of program evaluation contexts. 
These include the formative evaluation of a doctoral program in educational leadership 
(1990) and a baseline needs assessment of a high school academy located in a reservation 
community (1991).) This does not involve the massive subject pools required for the 
quantitative indicators: in fact, five to ten such judges is more than ample. The important 
thing is to get a balance (even one apiece) of such "experts" from a variety of perspectives, 
as illustrated in Figure 2, page 5. 

A number of experts in survey construction and research procedures have 
enthusiastically endorsed such "expert judge panel" pilot testing. DeVellis (1991) has 
pointed out the contribution that such expert judges can make in terms of assessing the 
validity of the instrumentation: 
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... haying experts review your item pool can confirm or invalidate your 
definition of the phenomenon. You can ask your panel of experts (e.g., 
colleagues who have worked extensively with the construct in question or 
related phenomena) to rate how relevant they think each item is to wliatyou 
intend to measure (emphasis in original text; page 75). 



Figure 2. 

Various Perspectives of "Expert Judges" 
in Instrumentation Pilot-Test Process 



Content Expert 
(e.g., professor) 



X 



(if possible) Survey 
Design Expert 
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Expert- Judge Validation Input 
into Refinement of Instrument 




(if needed) Key 
Community & 
Local Leaders 



What form should the collection of suggestions from pilot judges take? The good 
news for the busy, schedule-constrained teacher is: whatever is most convenient! These 5 
to 10 expert pilot judges may provide their thoughts individually in person; in written form 
at their convenience; on the telephone; or in a new (to education) and maximally efficient 
format known as a focus group. 

According to Richard Krueger (1988), a focus group is a small-group (4-12 
subjects) interview in which a relaxed setting is established beforehand by the interviewer, 
who indicates that all thoughts, opinions and feelings are valued. The objective is to get the 
participants interacting with, and reacting to, one another as well as to the interviewer. As 
Krueger has indicated, this is much like the opinion revision and refinement we do in 
everyday life in response to the thoughts and comments of our friends and colleagues. 
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The efficiency in using a focus group format for survey piloting is that quite often, 
expert judges make the exact-same comments and recommendations regarding a survey - 
whether their thoughts are solicited individually or in a group setting. The busy teacher can 
save much time and cost by bringing together his/her pilot subjects for a single one-hour 
session and obtaining all of their individual feedback in this limited time frame. 

Perhaps the best argument for using the focus group approach is the valid and 
reliable information it provides regarding the "fit" of the instrument for its intended locally 
appropriate circumstances. As pointed out by Morgan (1988): 

We are rapidly reaching a point at which most general population surveys 
consist entirely of items that have never been validated outside the confines 
of other surveys ... The most obvious way that focus groups can assist in 
item and scale construction is through providing evidence of how the 
respondents typically talk about the topic in question, a goal that is often 
summarized as learning their language on a topic. A more important use for 
preliminary focus groups is to ensure that the researcher has as complete a 
picture of participants' thinking as possible. This is especially important for 
making sure that issues that might have been ignored in an ousider's 
inventory of the topic are included, (page 34) 

Once the teacher has selected the handful of pilot judges and the way in which this 
pilot review will be conducted with them (e.g., individually; focus group; written 
comments on survey draft), the final decision to be made is how to compile their feedback. 
Whether through note-taking in an interview setting, receiving individual written 
comments, or tape-recording, the feedback will eventually have to be sorted out and 
distilled into a concise and usable fashion in order to produce a refined survey draft. 

One such procedure which "tells the story at a glance" is the matrix method. 
Originally developed and creatively illustrated by Miles and Huberman (1984), the matrix 
or table shell is a convenient grid which arrays the summarized comments in usually no 
more than a page or two. Bria and Dereshiwsky (1990) applied the Miles and Huberman 
matrix to such a pilot-test grid. It can be prepared by the teacher in "worksheet" format 
beforehand and used to sort, summarize and compile the pilot judges' feedback and is 
depicted in Figure 3, page 7. 

This process, including an actual example of such a pilot-test matrix, will be 
illustrated in detail in the following subsections. The author will use her own development 
and refinement of a mailed questionnaire to evaluate the 1992 Arizona Leadership Academy 
(ALA) to demonstrate the piloting process, matrix generation, and refinement of the 
survey. First, a brief overview of the nature and goals of the ALA will be provided. This 
will be followed by a description of the author's specific activities in generating and pilot- 
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testing the initial draft of the survey. The matrix of pilot judges' responses, along with the 
revised survey instrument, will be illustrated and discussed 



Figure 3. 
Sample General Pilot-Test Matrix 



Survey Subsection 


Pilot Judges' Comments 


Researcher Action Taken 


Instructions to Subjects 


Summary descriptive 
phrases and relative 
frequency counts 


Summary descriptive 
phrases and relative 
frequency counts 


Survey Section I (e.g., 
demographic items) 


Summary descriptive 
phrases and relative 
frequency counts 


Summary descriptive 
phrases and relative 
frequency counts 


Survey Section n, etc. 


Summary descriptive 
phrases and relative 
frequency counts 


Summary descriptive 
phrases and relative 
frequency counts 



An Example of a "Self-Made" Survey: 
Evaluating the Effectiveness of the 1992 Ariz ona Leadership Academy 

Background Information 



The 1992 Arizona Leadership Academy consisted of five days (June 15 - 19, 1992) 
of on-site large- and small-group activities for principals and teachers ("teams") from 
throughout the state. The goals of the academy experience, according to its faculty 
members and program planners, included focus on increased student academic 
achievement; professional growth; increased statewide and regional networking; improved 
site-based decision making; and facilitation of school planning and change. The first two 
days of the Academy were designed for principals only, to provide them with an 
opportunity for advance team building and facilitation activities, as well as to exchange 
ideas and concerns with their peers. These principals were subsequently joined by their 
team members on the third day. In addition to the specific goals outlined above, the teams 
were expected to develop and refine a specific plan intended to meet a locally appropriate 
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goal that they would then take back with them and begin to implement after the conclusion 
of the Academy. 

Held on the campus of Northern Arizona University, the ALA was conducted under 
the leadership of the Arizona State Department of Education. ALA program planners and 
faculty members consisted of a balance of State Department officials, local administrators 
and teacher-leaders, and university professors. The author of this paper was designated as 
the evaluator of the 1992 ALA and worked closely with all three groups in the planning and 
implementation of the evaluation. A total of approximately 400 participants (faculty 
members, program planners, team leaders and team members) took part in the 1992 ALA. 

Survey Design and Tnirial C onstruction 

Prior to drafting the survey instrument that ultimately became the primary means of 
data collection for the 1992 ALA evaluation, the author met with ALA program planners at 
an initial planning meeting for the Summer 1992 events and activities. In particular, she 
communicated closely with the Director of the 1992 ALA, Dr. David A. Wayne, in oider to 
identify his objectives and needs for specific information regarding the assessment of 
Academy effectiveness. The Director also provided the evaluator with extensive written 
documentation on the program, including predetermined published objectives and 
participants' goal statements as written during the Academy experience. These various 
perspectives and sources of information constituted a multimethod approach to the survey 
design and construction, which in turn served to strengthen the validity of the survey 
design process by approaching it from multiple viewpoints (Brewer and Hunter, 1989). 

Based upon these multiple sources, the author/evaluator identified the following 
informational needs to be included in the survey: 

1 . The ALA Director wished to determine if perceptions of ALA program effectiveness 
differed by the following subgroups of participants: 

a) first-time attendees vs. returnees; 

b) faculty members/program planners, vs. team leaders, vs. team members; 

2. The desired way to collect this information was via a mailed survey; 

3 . The survey was to consist of both demographic items and open-ended items; 

4. The open-ended items were intended to obtain participants' impressions of ALA 
effectiveness with respect to their: 

a) goals for improved student learning; 

b) goals for their own professional development; 

c) any other goals; 
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5. These three goals were to be assessed according to the following three dimensions 
or "time frames:" 

a) what they hoped to learn prior to the start of the ALA; 

b) how they are now applying what they actually learned during the ALA; 

c) how the ALA might do a better job with each goal in the future. 
Appendix A contains the author's initial draft of this survey, preceded by a cover 

letter to accompany its mailing to the pilot judges. (The author FAXed these materials to the 
ALA Director at the State Department, who in turn assumed responsibility for typing them 
on letterhead and mailing them to recipients.) 

In addition, the author/evaluator wished to facilitate the work of the pilot judges by 
preparing a "worksheet" for them to use in providing their feedback on the survey draft. 
This worksheet appears in Appendix B. 

Pilot-Test Procedures and, RgSIlliS 

A packet containing the cover letter, survey draft and worksheet was mailed to each 
of ten (10) pre-identified pilot judges. Seven (7) usable pilot- test comment packets were 
received by the specified return deadline. As is typical in qualitative data collection 
procedures, a veritable wealth of in-depth, revealing information resulted from this 
process. The pilot judges not only responded in detail on the worksheets, but they also 
extensively annotated the survey draft itself. 

In rereading, clustering and classifying this feedback, the author/evaluator 
discovered that the individually provided comments converged to a great degree. Thus, she 
was able to generate a matrix from these responses (Figc-e 4, pages 10-13). The numbers 
in parentheses following individual summary phrases indicate how many pilot judges 
independendy made the same suggestion, in cases of multiple mentions. Where no number 
appears parenthetically, only one pilot judge made the particular comment or suggestion. 
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Figure 4. 

Pilot-Test Matrix of Expert Judges 1 Comments: 
1992 Arizona Leadership Academy Evaluation 



Survey Subsection 


Pilot Judges 1 Comments 


Researcher Action Taken 


Demographics 


* Add #2: "In response to 


* Included question on 




Item# 1, above, lam 


primary professional 




primarilv identified as ... " 


affiliation 




(4): 






(a) Teacher, 






(b) Administrator; 






(c) Parent; 






(d) School district patron; 






(e) State Dept staff member, 






(f) University faculty 






member, 






(g) Mental health services 






provider, 






00 Bost! member, 






(i) Other (please specify) 






* Add question on "primary 


* Included question on 




location of residence" (4): 


primaiy location of 




(a) Rural; 


residence 




(b) Urban; 






(c) Suburban 






* Ask respondents to 


* Included an OPTIONAL 




identify own school district 


question asking participants 




(4) 


to reveal their district IF they 




* DON'T ask respondents to 


chose to do so 




identify school district (3) 
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Demographics, con't. 


* Add question on "school 


* Included mixtion nn 




lever 1 (2): 


school level 




fa) High school; 






(b) Junior high school; 






(c) Elementary school 






* Ask respondent to identify 


* Included auestion on AT A 




own role as part of the 


role 




school team (e.g., leader vs. 






member) (2) 






* Ask size of team 


* Included auestion on size 




* Ask length of time that the 


of team 




team has been functioning as 


* Included question on 




a team 


length of time as a team 




* Include a question on 


* Didn't include in this 




cultural/language diversities 


initial survey; possible 




to match Arizona student 


inclusion in planned follow- 




demographics with 


up site visits to participating 




leadership demographics 


ALA schools 
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Question #1 


* OK as is (2) 

* Improve format (more 
spacing) 

* Was not asked to state 
goals: such a question might 
be misleading 

* Goal statements too 
limiting (different schools 
are at different levels of 
leadership) 

.Let suojects state own 
goals 


* Added spacing 

* If this is the case, it should 
be indicated as such in the 
actual survey responses 

* Added a "team building" 
goals dimension to final 
survey draft 

* Open-ended nature of 
present survey items should 
already allow for this 
possibility 


Question #2 


* OK as is (clear and brief) 
(2) 

* Add "team building with 
staff 1 as a goals dimension 

* Liked questions on how 

A 1 A laommrrc j 1L ._ . 

r\L4t\ learnings are currently 
being applied; "It's 
important to apply learnings 
to behaviors" 


* Added a "team building" 
goals dimension to final 
survey draft 


Question #3 


* OK as is; "connects 
rccommenaauons to goals 
each participant/team 
member has set [for 
him/herself]" (4) 


* No changes implied 


Question #4 


* OK as is; "should yield 
some good responses for 
banners" (4) 

♦Start with this item; "it's 
good to identify the positive 
elements" 


* Decided to retain 
demographics as first 
section and keep this item as 
is 
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V^UCoUUn frj 


* Uiv as is 

* Combine questions #3 
and #5 


* Kept as separate questions 
(#3 is more specific, while 
#5 is general) 


Other/Miscellaneous 
Recommendations 


* Identify self as an 
independent evaluator 

* DONT include 
confidentiality clause ("How 
else can people improve if 
they are not given direct, 
open and honest feedback? 
We do this every day with 
students in our schools, 
why not with adults.") 

* Give assurances of 
confidentiality (report will 
not contain names, etc.) 

* Inform subjects of who 
will be conducting the 
follow-ups 


* Independent evaluator 
status prominently 
highlighted in accompanying 
cover letter 

* Confidentiality assurance 
retained in interests of 
protection and candor of 
subjects' responses 

* Added information 
identifying self as the 
follow-up (on-site visitation) 
evaluator) 


Other/Miscellaneous, con'L 


* Add a "report card" where 
respondents are asked to 
grade each of the five pre- 
stated goals of the ALA 
(from mission statement) 

* Add a dimension 
regarding what AT .A 
participants hoped to achieve 
prior to their actual Academy 
experience 


* "Report card" rating item 
added as survey item 

* "Hoped-for" goals added 

aa him uiLllCIlolun Ul ail 

open-ended items 
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Discussion of Pilot-Test Results 



As can be seen from this matrix, not every suggestion is automatically adopted by 
the survey developer; hence, the last column (researcher action taken). For one thing, this 
matrix reveals two areas where there were diametrically opposed opinions on the part of the 
pilot judges. These had to do with the desirability of asking respondents to identify their 
school districts and the assurances of respondent confidentiality. In the first case, the 
survey developer chose the "middle ground" of including a district identifier item BUT 
making it OPTIONAL in nature. In the second, a clear decision had to be made: in this 
instance, following recommended practice and assuring confidentiality to help protect 
subjects' anonymity and encourage complete candor in their responses. 

In another example of researcher discretion, it was decided not to adopt the 
recommendation on cultural and language diversities. The survey ended up being 
considerably lengthened as a result of the various suggestions for added demographic and 
open-ended items; this was the primary reason. In addition, since a series of on-site follow- 
up interviews had already been planned with the Director of the ALA, it was felt that this 
cultural diversity information could be more readily collected as part of the follow-ups. 

The final post-pilot draft of the survey appears in Appendix C. The interested reader 
is referred to Dereshiwsky (1993) for the results of the comprehensive qualitative 
evaluation of the 1992 Arizona Leadership Academy. 

Concluding Comments 

As teachers, we know the sense of satisfaction in empowering our students to take 
responsibility for all phases of their learning experience. Learners of all levels tend to enjoy 
"doing it all myself 1 when it comes to their own individual styles, readiness and needs to 
achieve. 

By the same token, teachers can attain the same positive benefits in empowering 
themselves to trust their own judgments in constructing surveys and tests that are right for 
them. The preceding example has illustrated how tapping a small number of pre-picked 
expert judges can result in rich and valuable information for refining a personally developed 
survey instrument. Such procedures are universally applicable to tests and surveys of 
virtually any topic area - and above all, fairly easy to apply. In trusting their expertise 
regarding what they need to measure, classroom teachers can more effectively take charge 
of their instructional environments - and become more greatly empowered professionals 
themselves in the process. 
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Appendix A. 

Pilot-Test Cover Letter and Initial Draft of Survey 



IS 



(DRAFT OF fl OVRR LETTER TO BE MATT .HP TO PTT .OT-TF,ST JUDGES^ 



(Name, address, date, etc.) 
Dear {Name of Recipient}: 

I have been designated as the evaluator of the Arizona Leadership Academy which was 
conducted during Summer 1992. I will be working closely with Dr. David Wayne of the Arizona 
State Department of Education in this activity. 

During the initial phase of this evaluation, we plan to mail an open-ended questionnaire to a 
sample of participants and program faculty. Our goal is to obtain their opinions, feelings and 
perceptions as to how the Leadership Academy has impacted important educational outcomes. 

You have been selected as an expert judge whose comments on the pilot draft of this survey 
would be most valuable. Would you please take a few moments to review this questionnaire and 
to provide feedback to me on the enclosed separate form? Please return your comments to me in 
die enclosed stamped, self-addressed envelope by {David, please insert ten (10) business days 
from the date that you issue the letter} . Or if you prefer, you may FAX them to me at the number 
provided below. 

In addition, we hope to follow up this initial survey with interviewing with a select smaller 
number of schools whose team members participated in the Academy. Our goal in these site 
visitations is to obtain more in-depth information from participants and their staff on the long- 
term impact of the Academy. If you would be interested in having your school participate in such 
on-site interviews, please indicate on the enclosed form. 

Your opinions on the initial draft of the Academy evaluation survev will be most valuable. 
Thank you very much in advance for your assistance. (If you should have any questions, please 
feel free to contact me directly at the numbers and/or address provided below. I will be back on 
campus on 8/26/92.) 

Sincerely yours, 



Mary L Dereshiwsky, Ph.D. 
Assistant Professor, 
Educational Leadership & Research 
Center for Excellence in Education 
Northern Arizona University 
Box 5774 

Flagstaff, AZ 8601 1-5774 

phone: (602) 523-1892 or -261 1 (E-mail messages) 

FAX: (602) 523-1929 
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(PILOT TEST DRAFT! 
INITIAL F.VAT.TTATTON OF TMPACT OF ARTZONA T F ApiTft fiHTP ACADEMY 



This survey is intended to obtain your thoughts, feelings and perceptions regarding the 
impact of the Arizona Leadership Academy on significant educational outcomes Your 
comments will be very helpful to us as we look at how the Academy has operated as well as anv 
ways it can be improved in the future. 3 

Please answer each question below as fully and candidly as possible If you need 
additional space, please feel free to use the back of the sheet and/or attach additional sheets of 
paper , indicating which question (number) is being continued. ALL RESPONSES WILL BE 
KEPT P STOCTLY CONFTPFrNTTAT » YOUR NAME AND/OR SCHOOL LOCATION WILL 
^BEATTACHED TO ANY INDIVIDUAL BESEQHSB&. THEREFORE, THERE ARE 
NO "RIGHT OR WRONG" ANSWERS. IN EXCHANGE, WE HOPE THAT YOU WILL 
FEEL FREE TO BE TOTALLY r AND TP IN YOUR RESPONSES. Thank you! 

Background Information - Please circle the QNJ£jesponse that BEST describes you. ' 
1 . My role in the Academy was: 

(a) Participant 

(b) Faculty Member 

(c) Program Planner 

(d) Other (please specify) 

•2. This was my first experience in the Arizona Leadership Academy: 

(a) Yes; 

(b) No. 

IF NO, how long have you been involved in the Academy? (blank line to fill in) 

*** : NOTE TO fflBQES: Should we ask them to identify their school/district? Or would that 
be too "threatening" despite the above assurances of complete confidentiality? 

JUDGES: Any OTHER demographic info which we should ask?? 

Evaluation Questions: Please answer each of the following questions as completely and candidly 
as possible. Please use the back of the page or additional sheets of paper if you need more room. 

1. Please identify three (1) goals that you hoped the Arizona Leadership Academy would 
help you accomplish in EACH of the following areas : 



21 



a) in terms of improved student learning outcomes: 



b) in terms of your own professional development : 



c) other goals (please specify): 



How are you applying what vou learned from the Arizona Leadership Academy with 
respect to: 

a) your goals for improved student learning? 



b) your goals for your own professional development? 



c) any other goals you identified in lc, above? 



How can the Arizona Leadership Academy do a better inh in ftg f mw 0 f helping its 



participants meet their: 
a) goals for improved student learning? 



b) goals for participants' professional development 1 ? 



b) any other goals? 



Please identify three (3) things that vou would NOT chang* a bom the Arizona Leadership 
Academy (e.g M activities, events) and why: 
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5. Please identify three (3) things that you think should be changed nr impro^ in the 
Arizona Leadership Academy in the future, and why : 



THANK YOU VERY MUCH for sharing your thoughts and opinions with us! 
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Appendix B. 
Pilot Judges 1 Worksheet 
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pilot judges: rnMiuFMT SHEEI 



NOTE TO PTLQT JTJPCtF,S : Please share your thoughts and ideas regarding the initial draft of 
£ 6 me tf doT Cy ' ^ ^ ^ l ° ^ 00 itSClf (& 1 f ° rget 10 retUni U 

Your ideas are valuable and will be very helpful to us in revising this survey! Thank you! 



Opening comments and instructions to subjects: 



Demographic section: 



Open-ended question #1: 



Open-ended question #2: 



Open-ended question #3: 



Open-ended question #4: 



Open-ended question #5: 



SUGGESTIONS???? 



I would like for my school to participate in the in-depth on-site future interviews: 

(a) No; 

(b) Yes. 



please provide your school's name & addlSSS, as well as the n ame, telephone number 
and FAX of a contar.t pergrffl; 



(lines for name, address, phone, FAX) 



THANK YOU AGAIN for your help in revising this draft of the evaluation of the Arizona 
Leadership Academy! 



Appendix C. 
Final (Post-Pilot) Draft of Survey 
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INITIAL EVALUATION OF IMPACT OF ARIZONA LEADE BSHIE ACADEMY 



This survey is intended to obtain your thoughts, feelings and perceptions regarding the impact of 
the Arizona Leadership Academy on significant educational outcomes. Your comments will be 
very helpful to us as we look at how the Academy has operated, as well as any ways it can be 
improved in the future. 

All individual responses will be read, analyzed and summarized by an independent evaluator (Dr. 
Mary I. Dereshiwsky, Assistant Professor, Educational Leadership & Research, Northern 
Arizona University) and doctoral student assistants working under her supervision. 

Please answer each question below as fully and candidly as possible. If you need additional 
space, please feel free to use the back of the sheet and/or attach ariflfoonal sheets of paper , 
indicating which question (number) is being continued. ALL RESPONSES WILL BE KEPT 
STRICTLY CONFIDENTIAL. YOUR NAME AND/OR SCHOOL LOCATION WILL 
BE ATTACHED TO ANY INDIVIDUAL RESPONSES. THEREFORE, THERE ARE NO 
"RIGHT OR WRONG" ANSWERS. IN EXCHANGE, WE HOPE THAT YOU WILL FEEL 
FREE TO BE TOTALLY CANDID IN YOUR RESPONSES. Thank you! 



L Background Information - Please circle the QEEjesponse that BEST describes you. 



1 . My role in the Academy was: 

(a) Participant - Part of a School Team 

(b) Participant - Leader of a School Team 

(c) Faculty Member/Program Planner 

(d) Other (please specify) 
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My professional affiliation is (PLEASE CIRCLE ALT, that apply and STAR your PRIMARY 



affiliation): 


(a) 


Teacher 


(b) 


Administrator; 


(c) 


Parent; 


(d) 


School/District Patron; 


(e) 


Mental Health Services Provider, 


(f) 


University Faculty Member; 


(g) 


School Board Member, 


CO 


State Department Staff Member; 


(i) 


Other (please specify). 


(if applicable') My school is: 


(a) 


Elementary level; 


(b) 


Junior high school level; 



(c) High school level. 

(if a pplicable^ My school is located in the following area: 

(a) Rural; 

(b) Suburban; 

(c) Urban. 

Current approximate enrollment (students): 
OPTIONAL : Please identify your district and school : 
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6. This was my first experience in the Arizona Leadership Academy: 

(a) Yes; 

(b) No. 

IF NO, how long have you been involved in the Academy? 

7. (a) The size of my ALA team was: 

(b) Total length of time that we have been a team: 



IL Rating of Mission Statement! Attached you will find a copy of the ALA Mission Statement 
and Goals. It would be very helpful to know what specific parts of the program of each week 
were most helpful and should be kept, modified, etc*) 

(a) On the line to the left of each stated goal, please "grade" it on a scale of A 
- B - C - D - F regarding to what extent the Academy met or exceeded each of 
these goals. (***: DAVID, IS IT POSSIBLE FOR YOU TO INCLUDE ONE 
COPY WITH, SAY, A LINE TO THE LEFT OF EACH GOAL FOR THEM TO 
GRADE IT?) 



(b) ^ What specific programs, events, activities, decisions, plans, new 
relationships, etc., are being carried out in your cun-ent professional situation as a 
result of your involvement in ALA? 



(c) In what spec ific ways are these having an imnaci (e.g., for students? the 
community? parents? team? staff? yourself personally? the school as a whole?) 



(d) What specific evidence do you have to document a nd measure this imp act (e.g., 
reports, news articles, announcements, personal testimony, studies, statistics)? 



HL Evaluation Questions: Please answer each of the following questions as completely and 
candidly as possible. Please use the back of the page or additional sheets of paper if you need 
more room. 



1. Please identify thr ee (3) goals that you hoped the Arizona Leadership Academy would 
help you accomplish in EACH of the following areas : 

a) in terms of improved student learning outcomes: 



b) in terms of your own professional development: 



c) 



other goals (please specify): 



d) Prior to the Academy, you were asked to state a goal that you hoped to achieve. 
What was this goal? 



e) How would you assess the impact of the Academy in helping you to attain this 
goal (part d, above)? 



How are you applying what vou learned from the Arizona Leadership Academy with 
respect to: 

a) your goals for improved student learning? 



b) your goals for your own profess ional development ? 



c) 



team building with your staff? 



d) any other goals you identified in lc, above? 



How can the Arizona Leadership Academy do a better job in the future of helping its 
participants meet their: 

a) goals for improved student learning? 



b) goals for participants' professional development? 
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c) team building with your staff? 



b) any other goals? 



Please identify three (3) things that you would NOT change about the Arizona Leadership 
Academy (e.g., activities, events) and wjr£ 



Please identify three (3) things that you think should be changed or improved in the 
Arizona Leadership Academy in the future, and how : 
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The independent evaluator of the Arizona Leadership Academy would like to continue 
this study with a series of followup visits and interviews to school sites whose staff 
participated in the Summer 1992 ALA. A series of confidential individual and small- 
group interviews of administrators, teachers and staff members is planned. The purpose 
of these visits and interviews is to continue to evaluate the long-range impact of ALA 
and the extent to which participants (and their peers) are applying what they have 
learned in their day-to-day educational job duties. 



Dr. Dereshiwsky would be happy to answer any questions that you may have about this 
followup phase of the study. She can be reached as follows: 

Ehfilis: (602)523-1892or-26U (messages) 

(602) 523-1929 

address: Mary L Dereshiwsky, PhD. 
Assistant Professor, 
Educational Leadership & Research 
Center for Excellence in Education 
Northern Arizona University 
Box 5774 

Flagstaff, AZ 8601 1-5774 

I would like for my school to participate in the in-depth on-site future interviews: 

(a) No; 

(b) Yes. 

IF YES, please provide your school's name and address, as well as the name, telephone number 
and FAX of a contact person* 



THANK YOU VERY MUCH for sharing your thoughts and opinions 
with us! 
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ARIZONA LEADERSHIP ACADEMY . Advanced Level - June 15-19, 1992 
Mission: 

To assist school leadership teams in making a positive difference in their schools and communities 
in order to improve the education of the students of Arizona. 



Goals: 



1. Participants will have the opportunity to focus on schools and communities 
working together to better meet the needs of Arizona students. 



2. Participants will examine the role of professional growth and development for 
educators and educational trends as related to their school district, their school and 
their classrooms. 



3. School teams will be encouraged to develop a networking process with school 
teams from around the state, as well as regionally, in order to support their local 
efforts. * 



4. Participants will focus on the new roles of leadership within schools, including 
shared decision making and site-based management or decentralized decision 
making. 



5. Participants and teams will learn and practice facilitation methods which will 
assist them, and their school colleagues, in successfully developing and 
implementing school planning and change. 



raac604.bjf, 10/92 
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